System and method for augmented reality inspection and data visualization

ABSTRACT

A 3D tracking system is provided. The 3D tracking system includes at least one acoustic emission sensor disposed around an object. The acoustic emission sensor is configured to identify location of a probe inserted into the object based upon time of arrival of an acoustic signature emitted from a location on or near the probe. The 3D tracking system also includes a first sensor configured to detect an elevation of the probe. The 3D tracking system further includes a second sensor configured to detect an azimuth of the probe.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of a priority under 35 USC 119 toIndian Patent Application No. 3014/CHE/2007, filed Dec. 18, 2007,entitled “SYSTEM AND METHOD FOR AUGMENTED REALITY INSPECTION AND DATAVISUALIZATION”, the entire contents of which is hereby incorporated byreference.

BACKGROUND

The invention relates generally to non-destructive inspection techniquesand, more particularly, to inspection techniques employing augmentedreality.

Inspection techniques are commonly used in a variety of applicationsranging from aircraft industry, health industry to securityapplications. Inspection of complex parts and structures generallyrequire immense inspector skill and experience. Borescope inspection isone of the commonly used sources of information for monitoring ofindustrial infrastructure due to easy access to in-service parts andreduced downtime. Condition based maintenance strategies on gas turbineand related systems rely heavily on data obtained from such inspection.Generally, probes that use long cables with display pendants have beenemployed for borescope inspection. However, once the probe is insertedinto a borescope inspection hole, minimal information about a locationand pose of a tip of the borescope is available to an operator. Trackingthe location and the pose reduces error in measurements and is veryimportant to accurately locate flaws and damages seen. Moreover, duringtip change scenarios, it is almost impossible to bring the tip to thesame location.

Thus a lot of the inspection is dependant on operator skill and issubjective. Accurate information about the borescope tip and pose alsoenables automation and control of an entire inspection process,beginning from inspection planning to guidance to damage reporting.

Therefore, a need exists for an improved inspection system thataddresses problems set forth above.

BRIEF DESCRIPTION

In accordance with an embodiment of the invention, a 3D tracking systemis provided. The 3D tracking system includes at least two acousticemission sensors disposed around an object. The acoustic emissionsensors are configured to identify location of a probe inserted into theobject based upon time of arrival of an acoustic signature emitted froma location on or near the probe. The 3D tracking system also includes afirst sensor configured to detect an elevation of the probe. The 3Dtracking system further includes a second sensor configured to detect anazimuth of the probe.

In accordance with another embodiment of the invention, an augmentedreality system for inspection within an object is provided. Theaugmented reality system includes a tracking system configured toidentify a 3D location of a probe inserted into the object. The 3Dtracking system includes at least one acoustic emission sensors disposedaround an object. The acoustic emission sensors are configured toidentify location of a probe inserted into the object based upon time ofarrival of an acoustic signature emitted from a location on or near theprobe. The 3D tracking system also includes a first sensor configured todetect an elevation of the probe. The 3D tracking system furtherincludes a second sensor configured to detect an azimuth of the probe.The augmented reality system also includes a microprocessor configuredto generate graphics and superimpose the graphics on the image capturedby the camera based upon the 3D location identified by the trackingsystem. The augmented reality system further includes a display unitconfigured to display an augmented reality image.

In accordance with another embodiment of the invention, a method of 3Dtracking within an object is provided. The method includes inserting aprobe into the object. The method also includes disposing at least oneacoustic emission sensor around the object. The method further includesattaching a first sensor and a second sensor to the probe.

In accordance with another embodiment of the invention, a method forforming an augmented reality image for inspection within an object isprovided. The method includes capturing an image via a camera. Themethod also includes identifying a location of a probe within the objectvia a plurality of acoustic emission sensors. The method furtherincludes determining elevation of the probe via a first sensor. Themethod also includes determining azimuth of the probe via a secondsensor. The method also includes generating graphics of the object. Themethod further includes registering the graphics on the image capturedbased upon the location, elevation and the azimuth determined to form anaugmented reality image.

DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a block diagram representation of an augmented reality imagesystem including a tracking system in accordance with an embodiment ofthe invention;

FIG. 2 is a block diagram representation of elements within the trackingsystem in FIG. 1;

FIG. 3 is a diagrammatic illustration of an exemplary borescope inaccordance with an embodiment of the invention;

FIG. 4 is a schematic illustration of an augmented reality image formedfor inspection of a gas turbine using the borescope in FIG. 3;

FIG. 5 is a schematic illustration of an exemplary display unit inaccordance with an embodiment of the invention;

FIG. 6 is a flow chart representing steps in an exemplary method for 3Dtracking within an object; and

FIG. 7 is a flow chart representing steps in an exemplary method forforming an augmented reality image for inspection within an object.

DETAILED DESCRIPTION

As discussed in detail below, embodiments of the invention include asystem and method for non-destructive inspection of an object. Thesystem and method disclosed herein generate an augmented reality imageusing an improved tracking system for inspection. As used herein,‘augmented reality image’ refers to an image that includes real worlddata superimposed with computer generated data. Non-limiting examples ofthe object include aircraft engines, gas turbines, steam turbines,diesel engines and a living organism.

Turning to the drawings, FIG. 1 is a high-level block diagramrepresentation of an augmented reality system 10 to inspect an object12. A tracking system 14 is employed to identify a 3D location andposition of a probe 13 inserted into the object 12. In a particularembodiment, the probe 13 includes a borescope or an endoscope. Thecamera 18 captures a view of the object 12 as a real image. In aparticular embodiment, the camera 18 captures a monocular view. Inanother embodiment, the camera 18 captures a stereoscopic view.Non-limiting examples of the camera 18 include a web camera, a videocamera, or a CCD camera. In another embodiment, more than one camera canbe used, for example two cameras could be arranged so as to providestereoscopic images. Non-limiting examples of the real image include avideo image and a still image.

The real image captured is used as a reference by a microprocessor 20that is configured to generate graphics corresponding to the real image.In an example, the graphics includes computer-aided design drawings ofthe object 12. The microprocessor 20 further superimposes the graphicson the real image based upon the 3D location identified by the trackingsystem 14 to generate an augmented reality image. Thus, the stereoscopicview obtained from the camera 18 is augmented with additionalinformation and provided to a user in real time. The additionalinformation may include, for example, text, audio, video, and stillimages. For example, in a surgical workspace, a surgeon may be providedwith a view of a patient including, inter alia, the view of the patientand an overlay generated by the microprocessor 20. The overlay mayinclude a view of the patient's internal anatomical structures asdetermined, for example, during a Computerized Axial Tomography (CAT)scan, or by Magnetic Resonance Imaging (MRI).

In another embodiment, the overlay includes a textual view of thepatient's medical and family history. The overlays may be displayed inreal-time. The augmented reality image includes the real image capturedby the camera 18 overlaid with an additional virtual view. The virtualview is derived from the microprocessor 20 and stored information, forexample, images. The augmented reality image also enables detection offlaws or cracks in the object 12. In an exemplary embodiment, themicroprocessor 20 includes a wearable computer. The microprocessor 20displays an augmented reality image on a display unit 22 but not limitedto, a personal digital assistant (PDA), pendant, an external computerand semi-transparent goggles.

It should be noted that embodiments of the invention are not limited toany particular microprocessor for performing the processing tasks of theinvention. The term “microprocessor” as that term is used herein, isintended to denote any machine capable of performing the calculations,or computations, necessary to perform the tasks of the invention. Theterm “microprocessor” is intended to denote any machine that is capableof accepting a structured input and of processing the input inaccordance with prescribed rules to produce an output.

FIG. 2 is a block diagram representation of elements 30 within thetracking system 14 in FIG. 1. The tracking system 14 includes at leastone acoustic emission sensor 32 disposed around the object 12 (FIG. 1).The acoustic emission sensor 32 is configured to identify location of aprobe 13 (FIG. 1) inserted into the object 12 for inspection. In aparticular embodiment, the acoustic emission sensor includes a diameterin a range between about 6 millimeters (mm) and about 12 mm. Thelocation is determined based upon calculation of a time of arrival of anacoustic signature emitted from within the object 12. In a particularembodiment, the acoustic signature is emitted via a single speaker ormultiple speakers disposed at a tip of the probe 13. A first sensor 34detects an elevation of the probe. In one embodiment, the first sensor34 includes a gravity sensor implemented in integrated micro-electricalmechanical systems (MEMS) technology and configured to detect theelevation based upon acceleration due to gravity. In another embodiment,the first sensor 34 includes a gyroscope implemented in integratedmicro-electrical mechanical systems (MEMS) technology and configured todetect the elevation based upon conservation of angular momentum. In yetanother embodiment, output of the first sensor 34 is integrated overtime to determine a position of the tip of the probe 13. In anon-limiting example, the acceleration is integrated twice over time todetermine the position.

The tracking system 14 further includes a second sensor 36 configured todetect an azimuth of the probe 13. In an exemplary embodiment, thesecond sensor 36 includes a magnetic sensor such as, but not limited to,a magnetic compass, configured to detect the azimuth in presence of amagnetic field. In an example, a solenoid is employed to apply amagnetic field to the magnetic sensor. In yet another embodiment, thesecond sensor 36 is a gyroscope that detects angular rotation rate alongthree orthogonal axes. In yet another embodiment, output of the secondsensor 36 is integrated over time to determine a position of the tip ofthe probe 13. In a non-limiting example, the acceleration is integratedtwice over time to determine the position.

FIG. 3 is a schematic illustration of an exemplary probe 13 (FIG. 1)such as a borescope 50 employed for inspection of an object such as, butnot limited to, a gas turbine. The borescope 50 includes a first sensor52 and a second sensor 54 disposed near a tip 56 of the borescope 50.Multiple ‘tiny’ speakers 58 are also disposed near the tip 56. It willbe appreciated that although multiple speakers 58 have been illustratedin FIG. 3, a single speaker may also be employed. The speakers 58 emitacoustic signals at various locations within the object that arecaptured by the acoustic emission sensor 32 (FIG. 2). In the illustratedembodiment, the first sensor 52 is a MEMS gravity sensor. The MEMSgravity sensor 52 measures acceleration/gravity along a sensitive axishaving direction 60. In one embodiment, when the borescope 50 is alignedvertically downward towards the earth in a direction 62, the sensitiveaxis 60 points vertically downward, resulting in a value ofacceleration/gravity equal to 1. In another embodiment, when theborescope 50 is tilted with respect to the vertical, the sensitive axisof the MEMS gravity sensor 52, along which the acceleration/gravity ismeasured, is tilted with respect to the vertical, and the MEMS gravitysensor 52 measures a value of acceleration/gravity less than 1.Accordingly, output of the MEMS gravity sensor detects an elevation ofthe borescope 50. In a particular embodiment, the MEMS gravity sensorincludes a diameter in a range between about 1 to about 4 mm. The secondsensor 54 may be an angular rate gyroscope implemented in integratedMEMS technology. The gyroscope senses a change in rotation of theborescope 50 and accordingly detects an azimuth of the borescope 50. Inone embodiment, the second sensor 54 includes a diameter in a rangebetween about 2 to about 8 mm.

FIG. 4 is a schematic illustration of an augmented reality image 70including real world data as in a gas turbine 72 and computer generateddata as in blades 74 interior to the gas turbine 72. The augmentedreality image 70 is obtained by superimposing graphics generated of theblades 74 on an image of the gas turbine including a casing captured bythe camera 18 (FIG. 1). In a particular embodiment, the image includes a2D image. The microprocessor 20 (FIG. 1) stores and registersinformation from the image of the gas turbine captured and generatesgraphics based upon the 3D location obtained from the tracking system 14(FIG. 1). The microprocessor 20 contains necessary software in order togenerate a graphical representation and an augmented reality image basedupon the image from the camera 18 and the generated graphicalrepresentation. Further, the microprocessor 20 contains a storage mediumin order to save and restore previously saved information.

In order to overlay an image, position and orientation of the camera 18with respect to the gas turbine 72, and the orientation of the gasturbine, need to be determined. As a result, it is desirable to know therelationship between two coordinate systems, a camera coordinate system(not shown) attached to the camera 18, and a coordinate system 78attached to the gas turbine 72. Tracking denotes the process ofmonitoring the relationship between the coordinate systems. Themicroprocessor 20 (FIG. 1) registers 3D location obtained from thetracking system 14 in a reference frame having the coordinate system 78of the gas turbine 72.

FIG. 5 is a schematic illustration of an exemplary display unit 100. Thedisplay unit 100 includes a handheld display commercially available fromGeneral Electric Inspection Technologies under the designation EverestXLG3®. The display unit 100 displays an image of blades 102 in aninterior of the gas turbine 72 (FIG. 4) and superimposed withinformation 104 generated by the microprocessor 20. Some examples of theinformation include a serial number of a blade, time of operation, andidentification of a crack. The display unit 100 also includes navigationbuttons 106 to select and edit the display.

As illustrated, a real view and a virtual view are blended. For example,the virtual view is provided as a transparency over the real view of thegas turbine 72. Registration between the real view and the virtual viewaligns the real and virtual views. Registration of the virtual viewincludes, inter alia, position, orientation, scale, perspective, andinternal camera parameters for each camera. Preferably, the internalcamera parameters such as, but not limited to, magnification aredetermined in a prior camera calibration procedure. The registeredvirtual view is aligned with the real image of the gas turbine 72 inreal time. In a particular embodiment, an operator carries the displayunit 100 which will provide him/her with an augmented reality view ofthe gas turbine 72. In an exemplary embodiment, the display unit 100 isof a “video see through” type.

“Video see-through” generates and presents an augmented reality world ata handheld display device such as display unit 100. The cameraintegrated with the display device is used to capture a live videostream of the real world. The camera 18 (FIG. 1) is located in relationwith the display unit 100 in such a way that it provides the same view,as an user would get by looking “through” the display device. The livevideo stream combined with computer-generated graphics is presented inreal-time at the display unit 100. Additional functionality includescamera zooming with output of the actual camera focal length. This willenable an accurate display of the computer-generated graphics correctlywhile zooming.

FIG. 6 is a flow chart representing steps in an exemplary method 120 for3D tracking within an object. The method 120 includes inserting a probeinto the object in step 122. One or more acoustic emission sensors aredisposed around the object in step 124. A first sensor and a secondsensor are further attached to the probe in step 126. In a particularembodiment, the first sensor and the second sensor are attached at a tipof the probe. In another embodiment, a single speaker or multiplespeakers are disposed at the tip of the probe.

FIG. 7 is a flow chart representing steps in an exemplary method 140 forforming an augmented reality image for inspection within an object. Themethod 140 includes capturing an image via a camera in step 142. Alocation of a probe within the object is identified in step 144 via oneor more acoustic emission sensors. In a particular embodiment, thelocation is identified by calculating a time of travel of an acousticsignal emitted from the tip of the probe to the acoustic sensors. Anelevation of the probe is determined via a first sensor in step 146. Anazimuth of the probe is further determined via a second sensor in step148. In an exemplary embodiment, the azimuth of the probe is determinedby applying a magnetic field to the second sensor. Graphics of theobject is generated in step 150. The graphics is registered on the imagecaptured based upon the location, elevation and the azimuth determinedto form an augmented reality image in step 152.

The various embodiments of an augmented reality system and methoddescribed above thus provide a way to achieve a convenient and efficientmeans for inspection. The system and method also provides for a guidedand enhanced insitu inspection & repair and foreign debris removal.Further, it provides a lower risk of forced outage due to improveddamage reporting.

It is to be understood that not necessarily all such objects oradvantages described above may be achieved in accordance with anyparticular embodiment. Thus, for example, those skilled in the art willrecognize that the systems and techniques described herein may beembodied or carried out in a manner that achieves or optimizes oneadvantage or group of advantages as taught herein without necessarilyachieving other objects or advantages as may be taught or suggestedherein.

Furthermore, the skilled artisan will recognize the interchangeabilityof various features from different embodiments. For example, the use ofa web camera with respect to one embodiment can be adapted for use witha pendant as a display unit described with respect to another.Similarly, the various features described, as well as other knownequivalents for each feature, can be mixed and matched by one ofordinary skill in this art to construct additional systems andtechniques in accordance with principles of this disclosure.

While the invention has been described in detail in connection with onlya limited number of embodiments, it should be readily understood thatthe invention is not limited to such disclosed embodiments. Rather, theinvention can be modified to incorporate any number of variations,alterations, substitutions or equivalent arrangements not heretoforedescribed, but which are commensurate with the spirit and scope of theinvention. Additionally, while various embodiments of the invention havebeen described, it is to be understood that aspects of the invention mayinclude only some of the described embodiments. Accordingly, theinvention is not to be seen as limited by the foregoing description, butis only limited by the scope of the appended claims.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A three-dimensional (3D) tracking systemcomprising: at least one acoustic emission sensor disposed around anobject, said at least one acoustic emission sensor configured toidentify a location of a probe inserted into the object based upon timeof arrival of an acoustic signature emitted from a location on or nearthe probe; a first sensor configured to detect an elevation of theprobe; a second sensor configured to detect an azimuth of the probe; animage capturing device configured to capture an image of said object;and a microprocessor, configured to: receive the captured image, theidentified location of the probe, the elevation and azimuth of theprobe, utilize the captured image to generate a corresponding real-timegraphics, and utilize the identified location, elevation and the azimuthof the probe to obtain a graphics of an object associated with saididentified location, elevation and azimuth of the probe; and superimposethe obtained graphics onto the real time graphics of the object.
 2. The3D tracking system of claim 1, wherein the acoustic signature comprisesan acoustic signature emitted via at least one speaker disposed at a tipof the probe.
 3. The 3D tracking system of claim 1, wherein the firstsensor comprises a micro-electro-mechanical systems (MEMS) gravitysensor configured to detect elevation based upon acceleration due togravity.
 4. The system of claim 3, wherein the micro-electromechanicalsyste (MEMS) gravity based sensor comprises a diameter in a range of 1millimeter (mm) to 4 mm.
 5. The 3D tracking system of claim 1, whereinat least one of the first sensor and the second sensor is disposed neara tip of the probe.
 6. The system of claim 1, wherein the at least oneacoustic emission sensors comprises a diameter in a range between 6 mmand 12 mm.
 7. The system of claim 1, wherein the second sensor comprisesa micro-electromechanical systems magnetic sensor configured to detectazimuth in presence of a magnetic field.
 8. The system of claim 7,wherein the micro-electromechanical systems magnetic sensor comprises adiameter in a range between 2 to 8 mm.
 9. The system of claim 1, thesecond sensor comprises one of; a magnetic compass and a gyroscope. 10.The system of claim 1, wherein the probe comprises a borescope.
 11. Thesystem of claim 1, wherein the object comprises a gas turbine.
 12. Anaugmented reality system for inspection within an object comprising: atracking system configured to identify a three-dimensional (3D) locationof a probe inserted into the object, the tracking system comprising: atleast one acoustic emission sensor disposed around the object, theacoustic emission sensor configured to identify a location of a probeinserted into the object based upon time of arrival of an acousticsignature; a first sensor configured to detect an elevation of theprobe; and a second sensor configured to detect an azimuth of the probe;a camera configured to capture an image of the object; a microprocessorconfigured to receive the captured image the identified location of theprobe, the elevation and azimuth of the probe, generate an augmentedreality image by superimposing on the image captured by the camera agraphics of an object determined based upon the location identified bythe tracking system; and a display unit configured to display saidaugmented reality image.
 13. The augmented reality system of claim 12,wherein the display unit comprises a handheld display.
 14. The augmentedreality system of claim 12, wherein the display unit comprises anexternal computer.
 15. The augmented reality system of claim 12, whereinthe image captured comprises one of: a two-dimensional (2D) or athree-dimensional (3D) image.
 16. The augmented reality system of claim12, wherein the real-time graphics comprises a plurality ofcomputer-aided design drawings of the object.
 17. The augmented realitysystem of claim 12, wherein the probe comprises a borescope.
 18. Theaugmented reality system of claim 12, wherein the object comprises a gasturbine.
 19. A method of 3D tracking within an object comprising:inserting a probe into the object; disposing at least one acousticemission sensor around the object; and attaching a first sensor and asecond sensor to the probe said first sensor and said second sensordetermining an orientation of said probe; determining a location of saidprobe based on a time of arrival of receipt of a signal detected by saidat least one acoustic emission sensor; capturing an image of the object;and generating an augmented image by obtaining a graphics of an objectassociated with said identified location and orientation of said probeand superimposing said obtained graphics onto said captured image. 20.The method of claim 19, further comprising attaching a plurality ofspeakers at a tip of the probe.
 21. The method of claim 20 wherein theattaching a first sensor and a second sensor to the probe comprisesattaching the first sensor and the second sensor near a tip of theprobe.
 22. A method for forming an augmented reality image forinspection within an object comprising: continuously capturing a seriesof images in real-time of the object; identifying a location of a probewithin the object via a plurality of acoustic emission sensors;determining an elevation of the probe via a first sensor; determining anazimuth of the probe via a second sensor; generating real-time graphicsof the object; and forming an augmented reality image by registeringgraphics on the captured real-time image of the object and graphics ofan object based upon the location, elevation and the azimuth of theprobe.
 23. The method of claim 22, wherein the identifying comprisescalculating a time of travel of an acoustic signal emitted from a tip ofthe probe to the acoustic sensors.
 24. The method of claim 22, whereinthe determining azimuth comprises applying a magnetic field to thesecond sensor.